Skip to content

fix(snapshot): authenticate exact incremental state - #17

Merged
farhan-syah merged 2 commits into
NodeDB-Lab:mainfrom
presempathy-awb:codex/upstream/snapshot-integrity
Jul 26, 2026
Merged

fix(snapshot): authenticate exact incremental state#17
farhan-syah merged 2 commits into
NodeDB-Lab:mainfrom
presempathy-awb:codex/upstream/snapshot-integrity

Conversation

@presempathy-awb

@presempathy-awb presempathy-awb commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Makes full snapshot restore and incremental snapshot application fail closed
against the exact authenticated state their manifest describes, and adds the
regression coverage needed to tell a complete, durable apply from a
superficially successful copy.

The previous implementation validated the manifest envelope but not the
relationships between the manifest, main-file pages, segment files, the target
header, and the follower's existing state. That left room for malformed or
incomplete artifacts to be accepted, for undeclared state to survive a restore,
and for a retry to authenticate stale plaintext cached before a failed apply.

Why these checks belong together

A snapshot is only valid if all of the following describe the same state:

  1. the authenticated manifest;
  2. the exact main-file page set and allocation floor;
  3. the active catalog and B+ tree roots;
  4. the exact active segment identities and their authenticated metadata;
  5. the destination mode, realm, base commit, and target commit;
  6. the durable journal actions that make the new state visible.

Validating any one of those in isolation is insufficient. A target header can be
structurally valid while naming a missing main page; a segment file can exist
under the expected name while carrying a different authenticated identity; and a
raw page rewrite can be correct on disk while a warm pager still holds plaintext
from the state that preceded it. Snapshot creation, staging, authentication,
publication, and cleanup are therefore treated as one ordered protocol.

The delta's page set is defined once

The base state contributes two page sets, and they are not interchangeable:

  • reader-visible — the data and catalog reachable set. This is the only set
    both sides of the protocol can compute, so it is what defines which pages a
    delta carries. Producer and follower now derive it with the same formula:
    target_reachable − base_reader_visible.
  • published — additionally the free-list chain and commit-history tree.
    apply_incremental carries both over from the follower's own header rather
    than from the producer, so those pages stay live across the apply and must
    never be overwritten. The producer cannot see them, so they can only ever be a
    write guard — never part of the delta's definition.

Conflating the two makes a healthy snapshot look malformed: subtracting
published when deciding which pages a delta should contain turns every page
the producer legitimately allocated over a follower-local free-list page into an
unexplained set mismatch.

Page reuse is not a defect

There is deliberately no producer-side page-reuse check. A page id below the
base allocation cursor proves nothing: a page that was free at the base commit
is legitimately reallocated for the target, and shipping it is safe precisely
because nothing base-reachable points at it. The cursor is an allocation
watermark, not a liveness boundary — rejecting on it fails every database that
has ever deleted anything, which is the steady state of the free-list design.

The collision that does matter — a page the follower's own free-list chain or
commit-history tree still hosts — is invisible from the producer. The follower
keeps both across an apply, and the base commit's recorded free-list root is
superseded writer metadata whose pages a later commit may already have recycled,
so reading it to guess would authenticate a page that is no longer a free-list
page at all. The follower owns that check and runs it before any byte reaches
main.db, reporting PagedbError::SnapshotBasePageReused with the offending
page id.

That error is deliberately not a corruption variant. Both states are internally
sound; they simply cannot be related by a page delta, and the remedy is a full
snapshot or a nearer base commit.

Full snapshot export and restore

Export fails if the source cannot supply every main page or segment file the
authenticated state requires, synchronizes copied files before publishing the
manifest, and records the exact manifest length so trailing data cannot be
silently accepted.

Restore requires an empty destination, so a snapshot is never merged into
unrelated pre-existing pages or segment files. Segment filenames are screened
during the copy and counted against the manifest before anything opens the
destination, so an undeclared sidecar is reported as artifact corruption rather
than as whatever error open-time recovery happens to raise.

Before the restored database is exposed, PageDB authenticates the restored
header, active root, reachable tree pages, segment manifests, and segment data
pages, and verifies the active segment set is exactly the set the manifest
declares: missing, renamed, corrupt, and extra undeclared files all fail.

Cleanup after a failure is scoped by recorded ownership rather than inferred
from an io::ErrorKind. A destination the operation created is removed; a
destination the caller pre-created is emptied but left in place.

Incremental export and application

Incremental export rejects a missing base commit and fails if any changed main
page or newly required segment file cannot be copied.

Incremental apply performs a complete validation pass before publication. It
requires follower mode and a matching realm; the follower's authenticated commit
to equal the declared base commit; a strictly advancing target commit; compatible
manifest fields; no trailing manifest bytes; complete delta records with no
truncation; one record per page ID; page IDs inside the target allocation bound
and never a reserved page; every target-reachable changed page to be present;
and every declared new segment to exist and authenticate as the expected
identity.

Base page sets and the base segment set are drawn from a single sample of the
published snapshot, so a concurrent gc_now — which Follower mode permits —
cannot split the base state across the comparison.

Pager cache coherency on retry

Raw incremental writes bypass the normal pager write path, so a validation read
during a failed apply can leave target-page plaintext in the pager cache. A
regression test forces a late failure through a corrupt segment, then corrupts a
target-root page and retries; before the fix the retry incorrectly succeeded
because validation reused the cached page rather than authenticating the bytes
now on disk.

Apply now resets cached main pages immediately after the raw delta writes and
before authenticating the target tree. The placement is exact: earlier would not
cover the raw writes, later would let stale plaintext influence validation.

Segment publication, tombstones, and recovery

The apply path stages and authenticates every promoted segment before
publication, computes the segment set reachable from the target catalog, and
emits journal actions for both promotion and tombstoning — so a target that
removes a segment still present in the follower directory does not leave the
physical store holding state the target catalog does not represent. Replay can
finish the same operation after an interruption.

Also covered: segment sets larger than one journal page, directory-sync failures,
deferred journal cleanup while readers remain pinned, replay after a promotion
failure poisons the handle, and serialization of concurrent applies before any
raw page write.

Regression coverage

The snapshot suite is the deliverable as much as the source change. Highlights:

  • incremental_round_trip_survives_page_reuse_below_the_base_cursor — a full
    delete/refill/export/apply cycle, asserting that reuse actually occurred so it
    cannot silently degrade into the append-only case.
  • incremental_snapshot_exports_reused_pages_below_base_cursor — parses
    pages.delta to prove at least one shipped record sits below the base cursor.
  • apply_incremental_refuses_to_overwrite_a_base_live_page_without_mutating_base
    — injects the base snapshot's own active root and asserts
    SnapshotBasePageReused names it, and that the base survives the refusal.

Two historical expectations were ruled out rather than transplanted:

  • the trusted mutable base cursor from the incoming manifest — the floor is now
    derived from authenticated follower metadata, and the cursor is not treated as
    a liveness boundary at all;
  • clearing an in-memory free-list on apply — the durable free-list root is
    authenticated and part of the published base state, so clearing it would now
    be incorrect.

Verification

On the merged head:

  • cargo nextest run -p pagedb --all-features --no-fail-fast: 492 passed,
    5 skipped;
  • the same suite under PAGEDB_INVARIANT_CHECKS=1: 492 passed, 5 skipped;
  • cargo fmt --all --check: pass;
  • cargo clippy -p pagedb --all-targets --all-features -- -D warnings: pass;
  • cargo doc -p pagedb --no-deps --all-features: no new warnings;
  • checklist-scaffold gate: pass.

The page-reuse regression was confirmed to fail against the pre-fix source and
pass after it.

Review focus

  1. whether the reader-visible / published split is the right boundary, and
    whether any other follower-local page set belongs in the write guard;
  2. whether cache invalidation sits at the correct raw-write boundary;
  3. whether segment promotion and tombstone journal actions stay replay-safe
    across every interruption point;
  4. whether exact segment-set validation is appropriately strict for both full
    restore and incremental apply;
  5. whether failure cleanup always leaves the destination reusable without hiding
    a durability error.

The change favors rejection over partial recovery whenever an artifact cannot
prove the exact authenticated target state.

presempathy-awb and others added 2 commits July 26, 2026 10:59
Incremental apply distinguished the base's reader-visible page set from
its published set only for the write guard, then reused the wider
published set to define which pages the delta must contain. That made
a producer's legitimate reuse of a page still held by the follower's
own free-list chain or commit-history tree look like a malformed
artifact instead of the true base/target mismatch it is.

Split the two page sets explicitly, report the real collision as a new
SnapshotBasePageReused error, restrict segment restore to segment-id
filenames, and record destination ownership so failed export/restore/
apply cleanup never deletes a directory the caller pre-created.
@farhan-syah
farhan-syah merged commit a812d42 into NodeDB-Lab:main Jul 26, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants